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EDP  AUDITS 


Electronic  data  processing  (EDP)  audits  conducted  by  the  Office  of  the  Legislative  Auditor  are 
designed  to  assess  state  government  operations.  From  the  audit  work,  a  determination  is  made 
whether  agencies  have  adequate  controls  in  their  data  processing  systems,  whether  agency  data 
processing  operations  are  accomplishing  their  purposes,  and  whether  they  can  do  so  with  greater 
efficiency  and  economy.  In  performing  the  audit  work,  the  audit  staff  uses  audit  standards  set 
forth  by  the  United  States  General  Accounting  Office  and  the  American  Institute  of  Certified 
Public  Accountants. 

Members  of  the  EDP  audit  staff  hold  degrees  in  disciplines  appropriate  to  the  audit  process.  One 
member  is  a  Certified  Information  Systems  Auditor  and  the  other  is  a  Certified  Public  Accoun- 


EDP  audits  are  performed  at  the  request  of  the  Legislative  Audit  Committee  which  is  a  bicameral 
and  bipartisan  standing  committee  of  the  Montana  Legislature.  The  committee  consists  of  four 
members  of  the  Senate  and  four  members  of  the  House  of  Representatives. 
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The  Legislative  Audit  Committee 
of  tine  Montana   Legislature: 

During  our  recent  review  of  tine  Information  Processing  Facil- 
ity and  selected  applications  for  fiscal  year  1984-85,  we  identified 
specific  areas  of  concern  relating  to  recovery  from  a  disaster  at 
the  central  computer  facility.  This  audit  report  contains  conclu- 
sions and  a  recommendation  concerning  disaster  recovery.  The 
agency  response  is  contained  at  the  end  of  the  report. 

We  wish  to  express  our  appreciation  to  the  director  of  the 
Department  of  Administration  and  her  staff  for  their  cooperation 
and  assistance. 


Respectfully  submitted. 


Scott  A.   Seacat 
Legislative  Auditor 
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SUMMARY  OF  RECOMMENDATIONS 


Many  agency  operations  are  dependent  on  their  ability  to  process 
information  using  the  state's  central  computer.  We  found  the  state 
is  not  well  prepared  in  the  event  of  a  temporary  or  permanent  loss 
of  the  central  computer. 

Specifically,   the  report  notes  that: 

-  The    Information    Services    Division     (ISD)     does    not    have    an 
update  plan   for  dealing   with  a  disaster. 

-  Most    state    agencies    do    not    have    a    written,    tested    plan    for 
disaster   recovery. 

-  The    state    has    not    determined    which    applications    will    run    at 
ISD's  backup  computer  facility. 

We  recommend  the  Department  of  Administration  work  with  user 
agencies  to  develop  workable  disaster  recovery  plans.  The 
department  has  concurred. 
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DISASTER   RECOVERY 

During  a  recent  audit  at  the  state's  central  computer  facility, 
we  noted  that  the  state  of  Montana's  capability  to  recover  from  a 
disaster  to  its  computer  operations  is  weak.  This  report  details 
our   review  and  contains   recommendations  for  improvements. 

OBJECTIVES  AND  SCOPE 

The  objectives  of  the  disaster  recovery  audit  were  to: 

1.  Determine    the    Information    Services    Division's    (ISD)    role 
in  disaster  recovery. 

2.  Determine  agencies'   role  in  disaster  recovery. 

3.  Evaluate     the     adequacy     of     agency     disaster     recovery 
plans. 

4.  Identify  critical  applications  of  state  agencies. 

5.  Assess    the   capability   of   the    state    in    recovering    from    a 
major  disaster. 

During  our  audit  we  interviewed  employees  of  ISD  and  per- 
sonnel at  19  state  agencies.  We  also  reviewed  documentation  to 
supplement  and  confirm  information  obtained  through  interviews. 

The  agencies  reviewed  were  limited  to  those  located  in  the 
Helena  area  known  to  have  computer  applications  on  ISD's  main- 
frame. Separate  EDP  audits  that  included  disaster  recovery  have 
been  conducted  at  Montana  State  University  and  the  University  of 
Montana. 


BACKGROUND 


The  Information  Services  Division  (ISD)  within  the  Department 
of  Administration  provides  data  processing  services  for  use  by 
state  agencies.  Processing  is  performed  on  an  IBM  3033  and  an 
IBM  4381  (mainframe  computers).  In  addition,  all  peripherals 
(tape    drives,     disk    drives,     printers,     etc.)     are    maintained    and 


supported  by  the  division.  State  agencies  "purchase"  ISO's 
services  to  process  their  applications.  These  applications  include 
the  Statewide  Budgeting  and  Accounting  System;  the  Payroll/Person- 
nel/Position Control  System;  the  Motor  Vehicle  System;  Fish, 
Wildlife,  and  Parks  Drawing  System;  and  Unemployment  Insurance 
Check  Writing  System.  A  listing  of  applications  considered  critical 
to  state  operations  by  the  19  agencies  reviewed  is  located  in 
Appendix  A. 

If  a  disaster  occurred  to  ISO's  computer  processing  center, 
the  majority  of  agency  applications  would  no  longer  be  able  to  be 
processed  on  the  state's  mainframes.  SRS,  which  relies  on  the 
mainframe  to  write  over  1,500  Medicaid  checks  weekly  and  6,500 
AFOC  checks  monthly,  would  have  to  resort  to  a  manual  system 
which  could  cause  long  delays  in  receipt  of  the  checks  by  those 
persons  depending  on  them.  The  Department  of  Fish,  Wildlife  and 
Parks  would  have  to  conduct  its  annual  big  game  drawing  by 
hand,  resulting  in  a  possible  loss  of  all  or  part  of  the  $2.1  million 
in  revenue  received  in  1985.  The  9,200  unemployment  insurance 
checks  delivered  weekly  would  not  be  written  by  an  automated 
system,  resulting  in  a  delay  in  recipients  receiving  over  $2.2  mil- 
lion. All-purpose  warrants,  which  number  approximately  3,189  per 
day,  would  not  be  delivered  in  a  timely  manner  to  the  various 
vendors  deserving  them.  Over  11,000  state  employees  would  not 
be  paid  every  other  Wednesday,  having  to  wait  while  the  Central 
Payroll  Division  manually  writes  every  check.  The  above  agencies 
would  have  to  hire  additional  personnel  to  process  these  applica- 
tions in  a  timely  manner. 

The  above  examples  show  the  possible  effects  of  a  major 
disaster  to  Montana's  computer  center.  They  also  emphasize  the 
need  for  disaster  recovery  planning.  Six  million  dollars  of  compu- 
ter equipment  was  recently  soaked  in  sewage  water  as  the  result  of 
a  flood  in  England.  The  equipment  was  totally  destroyed  as  was 
all  of  the  company's  documentation.  The  computer  staff,  who  had 
no    contingency    plan    in    case    of    destruction    of    its    hardware    and 


software,  was  forced  to  work  around  the  clock  for  three  months 
after  the  disaster  re-creating  lost  information.  The  disruption  of 
business  and  the  high  cost  associated  with  this  disaster  was  large- 
ly due  to  the  company  not  having  a  disaster   recovery  plan. 

A  disaster,  when  associated  with  a  computer  center,  relates 
primarily  to  the  disruption  or  destruction  of  computer  resources. 
Disasters  to  a  computer  facility  take  many  forms.  Minor  operation- 
al errors,  temporary  loss  of  power,  and  total  destruction  to  the 
computer  center  may  each  constitute  a  disaster.  The  key  to 
recovery  from  the  many  different  potential  levels  of  disruption  to  a 
computer  facility  is  a   recovery  plan, 

A  feasible  recovery  plan  should  be  a  formally  documented 
listing  of  the  following   topics: 

-  Data:       A    listing    of    each    off-site    master    file,     including    its 
date,   location,   and  procedures  for  updating. 

-  Software:       A     listing    of    the    location    and    arrangements    for 
off-site  backup  essential  to  processing   pertinent  applications. 

-  Hardware:     A  listing  of  hardware  required  to  process  required 
applications. 

-  Supplies:     A  listing  of  all  forms  and  supplies  stored  off-site. 

-  Documentation:      A    listing    of  off-site   copies   of   source   codes, 
application   run  manuals,   and  operator  manuals. 

-  Personnel:      A    listing    of   names    and    phone    numbers    of   perti- 
nent personnel. 

-  Facilities:     A  description  of  space  and  support  services. 

In  addition,  a  plan  should  cover  several  levels  of  disruption, 
be  tested  to  determine  the  reliance  which  may  be  placed  on  its 
workability,  and  be  stored  at  appropriate  off-site  locations. 

In  February  1986,  ISD  will  have  its  IBM  4381  located  at  the 
National  Guard  Armory  to  serve  as  a  backup  to  its  IBM  3033  in  the 
event  of  an  emergency.  This  computer's  primary  function  will  be 
to    run    the    Department    of    Justice's    Criminal    Justice    Information 


Network    (CJIN),    but   it   will   have  the  capability  of  running   selected 
agency  applications. 

ISD  and  those  agencies  who  process  applications  on  the 
mainframe  need  disaster  recovery  plans  outlining  the  above 
mentioned  topics.  These  plans  should  indicate  the  capability  of 
the  state  to  recover  from  a  disaster.  The  following  sections  detail 
our  findings  in   regard  to  the  extent  of  this  capability. 

ISP'S   ROLE 

ISD  has  determined  its  responsibility  in  regard  to  disaster 
recovery   is  to  provide  the  following: 

-  Hardware    and    systems    software    to    enable    state    agencies    to 
process  their  critical  applications. 

-  A  facility  to  house  the  necessary  hardware. 

-  Assistance    to    state    agencies    in    developing    feasible    disaster 
recovery  plans. 

ISD  presently  has  a  disaster  recovery  plan  outlining  its 
responsibility  in  the  event  of  a  disaster.  This  plan,  dated  July 
1981,  does  not  include  the  future  off-site  computer  facility  at  the 
Armory.  The  plan  addresses  the  resources  needed  to  recover  from 
a  disaster,  but  it  is  not  current  and  does  not  reflect  future  needs. 
It  also  does  not  specifically  define  levels  of  disasters  or  proce- 
dures needed  to  recover  from  each. 

ISD  personnel  have  stated  that  an  updated  disaster  recovery 
plan  has  not  been  a  high  priority  due  to  the  large  amount  of  time 
involved  in  developing  and  implementing  it.  The  belief,  that  the 
existence  of  an  off-site  facility  was  needed  before  a  workable  plan 
could  be  developed,  has  also  contributed  to  ISD's  present  outdated 
plan. 

AGENCIES'   ROLE 

To  determine  what  various  state  employees  believe  their  roles 
are  in   regard  to  disaster  recovery,   we  interviewed  personnel  of 


19  agencies.  Responses  to  our  questions  regarding  agencies'  roles 
were  varied.  The  following  are  examples  of  agency  personnels' 
perceived   roles: 

-  "Responsible  for  making  arrangements  to  recover  needed 
data." 

-  "Disaster   recovery   is   ISD's   role." 

-  "Responsible  for  getting   tilings   running  again." 

-  "To  ensure  disaster  recovery  provisions  are  in  place." 

-  "To  provide  the  same  level  of  services  as  prior  to  disaster." 

As  shown  by  the  above  examples,  many  different  beliefs  exjst 
regarding  agencies'  roles  in  the  event  of  a  disaster.  Based  on  our 
review  of  state  agencies,  as  well  as  ISD,  we  consider  the  following 
to  be  the  responsibility  of  state  agencies: 

-  Developing  a  feasible  disaster  recovery  plan  which  lists  the 
necessary  attributes  for  effective  recovery  of  critical  applica- 
tions. 

-  Requesting  ISD  to  backup  and  store  critical  software  and  data 
files  off-site. 


Disaster  Recovery  Planning 

As  stated  previously,  the  key  to  safeguarding  agency  applica- 
tions is  a  well-documented  and  current  disaster  recovery  plan. 
About  two-thirds  (12/19)  of  the  agencies  contacted  during  our 
audit  had  no  written  disaster  recovery  plans.  Many  state  applica- 
tions, such  as  Water  Rights,  Lands  and  Investments  Leases,  Big 
Came  Drawing,  Executive  Budget  System,  Adult  and  Juvenile 
Correctional  Information  Systems,  Mortgage  Portfolio  and  Account- 
ing, and  Uniform  Commercial  Code,  have  no  arrangements  for 
continued  operation  in  the  event  of  an  emergency.  Specific  con- 
cerns have  developed  because  of  the  present  lack  of  disaster 
planning. 


Communication 

Communication  between  ISD  and  state  agencies  in  regard  to 
the  need  for  agency  level  disaster  recovery  plans  could  be 
improved.  Our  review  indicates  that  agency  personnel  do  not 
understand  the  need  for  a  written  disaster  recovery  plan  nor  what 
should  be  contained  within  one.  ISD  has  recognized  this  need  but 
has  not  provided  sufficient  encouragement  for  the  development  of 
agency  plans. 

ISD,  as  the  major  "supplier"  of  data  processing  services  in 
the  state,  should  assume  the  responsibility  of  assisting  agencies  in 
the  development  of  workable  disaster  recovery  plans.  This  close 
assistance  would  ensure  the  plans  meet  the  needs  of  the  agencies, 
as  well  as  contain  necessary  information  for  ISD  to  begin  process- 
ing critical  applications  at  an  off-site  facility. 

Portability 

Because  of  the  dependencies  specific  applications  have  on 
other  resources,  most  state  agencies'  applications  are  not  "porta- 
ble." Portability  refers  to  the  ease  with  which  data  can  be  imme- 
diately processed  on  another  computer.  The  running  of  an  appli- 
cation involves  an  operating  system  and  general  software,  as  well 
as  appropriate  hardware.  "Portability"  ensures  all  of  these  are 
available  when  needed.  In  the  event  of  a  disaster,  state  agencies 
could  have  a  difficult  time  running  critical  applications  even  at 
ISD's  off-site  facility. 

To  ensure  the  effective  processing  of  agency  data,  specific 
dependencies  unique  to  applications  must  be  identified.  Once 
recognized,  necessary  support  can  be  prepared  and  ready  for 
processing  at  an  off-site  facility. 

PRIORITIZATION 

Certain  agency  applications  exist  which  are  critical  to  the 
operation  of  state  government.  In  the  event  of  a  disaster,  only  a 
portion  of  these  applications  would   be  able  to  be  run  at  the  state's 


off-site    facility.       At    this    time,     no    sole    state    entity    has    taken 
responsibility  for  the  prioritization  of  agency  applications. 

During  our  audit  we  asked  personnel  from  each  of  the  19 
agencies  contacted  to  identify  their  critical  applications  (see 
Appendix  A).  Sixty-three  applications  were  listed.  Although 
some  applications  appeared  to  be  less  critical  than  others,  each 
agency  believed  specific  legal  and/or  financial  ramifications  would 
occur  if  any  of  these  63  applications  were  not  run.  Because  of 
the  potential  implications  and  impacts  on  the  functioning  of  the 
state  as  a  whole,  a  single  entity  is  needed  to  prioritize  critical 
applications. 

Under  the  broad  authority  given  to  the  director  of  the  Depart- 
ment of  Administration  by  section  2-17-501,  MCA,  the  director 
should  assume  the  responsibility  of  prioritizing  applications.  The 
Data  Processing  Advisory  Council,  which  consists  of  fourteen 
department  directors,  or  their  designees,  whose  agencies  are  the 
state's  major  computer  users,  appears  to  be  a  feasible  body  to 
assist  the  director  in  prioritizing  of  state  applications.  The  coun- 
cil should  help  identify  which  applications  are  critical  to  state 
government  and  what  priority  each  would  have. 

Because  the  criticality  of  applications  to  state  operations 
would  vary  depending  on  the  timing  of  the  disaster,  specific 
priority  schedules  should  be  developed.  These  schedules  could 
detail  what  applications  would  be  run  if  a  disaster  occurred  at  a 
certain  time  during  the  processing  cycle  (week,  month,  quarter, 
etc.). 

TESTING 

Once  ISD  has  its  off-site  computer  facility  and  agencies 
develop  their  own  disaster  recovery  plans,  testing  will  become 
necessary  to  determine  the  workability  of  the  plans.  Each  element 
of  the  recovery  plans  should  be  tested  by  actually  running  the 
applications  at  the  off-site  facility.  Problems  with  the  plans  can 
be  corrected  during  testing  which  would  enable  more  timely  recov- 
ery   in    the   event   of  an   actual    disaster.      Testing    would    also    help 


ISD  determine  what  resources  would  be  needed  on  its  backup 
computer  for  each  application  and  which  users  can  be  accommodated 
concurrently. 

ISD  has  recognized  the  need  for  the  testing  of  agency  appli- 
cations at  the  off-site  facility.  The  division  has  also  recognized 
the  cost  of  testing.  At  this  time  agencies  have  no  specific  alloca- 
tions for  testing.  If  ISD  absorbs  this  cost,  computer  user  rates 
could  increase. 

SUMMARY 

The  state's  ability  to  recover  from  a  major  disaster  to  its 
computer  operations  is  weak.  ISD's  role  is  to  provide  the  facility, 
personnel,  and  equipment  necessary  to  run  agency  applications  in 
the  event  of  a  disaster.  ISD  will  soon  partially  fulfill  this  respon- 
sibility with  the  presence  of  its   IBM  4381   at  an  off-site  facility. 

ISD  needs  to  revise  its  disaster  recovery  plan.  Agencies  are 
not  meeting  their  responsibility  of  developing  feasible  recovery 
plans  for  individual  applications.  Agencies  should  request  ISD  to 
provide  them  assistance  regarding  the  development  of  recovery 
plans,  portable  applications,  and  finally  the  testing  of  these  plans. 
The  director  of  the  Department  of  Administration,  with  assistance 
from  the  Data  Processing  Advisory  Council,  needs  to  identify  and 
prioritize  critical  applications  to  keep  state  government  "running" 
in  the  event  of  a  disaster. 

RECOMMENDATION   #1 

WE  RECOMMEND  THE  DEPARTMENT  OF  ADMINISTRATION 
WORK  WITH  USER  AGENCIES  TO  DEVELOP  WORKABLE  DISAS- 
TER  RECOVERY   PLANS. 


APPENDIX  A 

The    following    is    a    listing    of    state    applications    which    have    been 
identified  by  the  agencies   listed  as  critical: 


1 .  Department  of  Administration 

-  Property  Accountability  and  Management  System 

-  Statewide  Budgeting  and  Accounting   System 

2.  State  Auditor 

-  Payroll/Personnel/Position   Control   System 

-  Warrant  Writer  System 

3.  Department  of  Commerce 

-  Section   8   Housing   System 

-  Travel   Promotion   System 

-  Mortgage  Portfolio  and  Accounting  System 

-  Nurse   Licensing   System 

-  Professional  and  Occupational   Licensing  System 

4.  Department  of  Natural   Resources  and  Conservation 

-  Water   Rights  System 

-  Water  Modeling  System 

-  Oil  and  Gas  Records 

5.  Fish     Wildlife  and   Parks 

-  Drawing   System 

6.  Governor's  Office 

-  Executive  Budget  System 

-  Legislative  Appropriation   System 

-  Revenue  Estimate  System 

-  Monthly   Reporting  System 

7.  Department  of  Highways 

-  Bid  System 

-  Gross  Vehicle  Weight  Proration  System 

-  Payroll 

-  Project  Estimates 

-  Billing  Voucher   Interim  System 

-  Statewide  Budgeting  and  Accounting  System 

8.  Department  of  Institutions 

-  Adult  Correctional   Information  System 

-  Juvenile  Correctional    Information  System 

-  Automated   Billing  Accounts   Receivable  System   (ABARS] 

-  Supply   Inventory  Monitoring  System 

-  Resident  Account  System 

9.  Department  of  Justice 

-  Driver  License  System 

-  Criminal   History 


-  Motor  Vehicle   Registration  System 

-  Criminal  Justice   Information   Network   (CJIN) 

10.  Department  of  Labor 

-  Unemployment  Insurance  Check  Writing 

-  JTPA  Check  Writing 

-  Unemployment   Insurance  Monetary  Determinations 

11 .  Department  of  State  Lands 

-  State  Trust  Land  Management 

-  Lands  and   Investments  Leases 

-  Nursery  Tree  and   Fire  Personnel 

12.  Department  of  Livestock 

-  Livestock   Brands 

13.  Secretary  of  State 

-  Uniform  Commercial   Code  System 

-  Corporate  Automated  System 

14.  Department  of  Social  and   Rehabilitation  Services 

-  Montana   Income  Maintenance  System   (MIMS) 

-  SRS  Client  Database 

-  Low   Income  Energy  Assistance  Program   (LIEAP) 

-  Accounts  Receivable  System 

15.  Supreme  Court 

-  Supreme  Court  Information  System 

16.  Legislative  Council 

-  Bills  Processing 

-  Journal   Processing 

-  Bill   Status  System 

-  Publication  System 

-  Publication   Distribution  System 

17.  Public  Employees'   Retirement  Division 

-  Public  Employees'   Retirement  System 

18.  Teachers'   Retirement  Division 

-  Teachers'   Retirement  System 

19.  Department  of  Revenue 

-  Income  Tax 

-  Natural   Resource  and  Corporation  Tax 

-  Welfare  Fraud   Investigation 

-  Child  Support  Enforcement 

-  Bad  Debt 

-  Accounts   Receivable 

-  Liquor 

-  Miscellaneous  Tax 

-  Motor  Fuels 

-  Property  Assessment 
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AGENCY   RESPONSE 


DEPARTMENT  OF  ADMINISTRATION 

DIRECTORS  OFFICE 


TED  SCHWINDEN.  GOVERNOR 


IITCHELL  BUILDING 


STATE  OF  MONTANA' 


406)449-2032 


HELENA.  MONTANA  59620 


October  11,  1985 


P!r/^T:\  "TH 


Scott  A.  Seacat 
Legislative  Auditor 
State  Capitol 
Helena,  Montana  59620 


•L^n/t  AoniTGS 


Dear  Scott: 

The  department's  Information  Services  administrator  and  his 
staff  have  reviewed  the  audit  report  and  recommendation  on 
disaster  recovery  and  we  concur  with  the  recommendation. 

The  Information  Services  Division  will  continue  to  work  with 
user  agencies  to  develop  a  workable  disaster  recovery  plan 
and  improve  communications  in  order  to  clarify  the  roles  of 
agencies  and  ISD.   We  plan  to  issue  written  guidelines  defin- 
ing the  mechanism  for  establishing  and  testing  a  transportable 
system  for  auxilliary  site  operation  by  February  1986.   The 
IBM  4381  should  be  operational  in  the  National  Guard  Armory 
by  then. 

Prioritization  of  the  state's  critical  applications  will  be 
addressed  and  resolved  prior  to  the  end  of  this  fiscal  year 
(June  30,  1986).   The  Disaster  Recovery  Plan  will  be  updated 
and  made  current  in  that  same  timeframe. 


We  appreciate  the  involvement  of  your  office  and  we  share  your 
concern  for  establishing  a  viable  disaster  recovery  capability. 


Sincerely 


■■i^^<:>iy 


ELLEN  FEAVER 
Director 


'  EOUAi.  OPPORTUNITy  EMPLOYER  ■ 


